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(54) PICTURE GENERATION METHOD 

(57)Abstract: 

PROBLEM TO BE SOLVED: To change the view point to 
observe a picture from various view point positions by using 
pictures taken in plural different photographing positions to 
generate a picture which may be obtained at the time of 
photographing in an assumed photographing position. 
SOLUTION: An inputted picture is stored in a storage area 
405 reserved in a storage area 400 on a storage device. 
Two sets of pictures photographed in two or more positions 
are inputted. Plural generated panorama pictures are 
divided into pictures of plural areas in accordance with 
objects in these pictures. Attribute information indicating the 
sequence of superposition synthesis to respective area 
pictures is defined. How the whole of panorama pictures 
should be distorted for each area picture is defined to 

generate distortions. The relative view point of an observer with the photographing position of a 
digital camera as the reference and the sight direction indicating the direction which the observer 
is looking toward are inputted. Respective divided area pictures are deformed based on 
calculated extents of distortion of these picture areas, and they are synthesized by superposition 
in accordance with attribute information to generate a panorama picture. 
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im&msi m&mi. 2. 3, 4, 5, 6tttci*7m 

HM»H1, 2, 3. 4. 5, 6£fc«7ie 

IB 2 ©x-r 
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[0005] (b) TView Interpolation for Image S 
ynthesis (1993) J , Computer Graphics Proc, 279~ 
288H, Apple Computer Inc. 
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[0 0 16] COiMc, *«E©!H*£jaWffiTtt, 

IK/ V 5 TB»fr 5 « 0 a? o fc»#B«*a*^fc-» l> 

[0017] cntctt), fie*sffi©<t^^. i&d&a, 
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[0018] ^fc, nma^ mn<Di$&wmiti&&'a> 

*«**C*©-e, «S»H«* % )Sffl«©^«©B« 
[0 0 19] *«M©IS»^J«35r»tefeV>T, ± 

40 m'V9"vM»*£jaw*aj®*, ±&v5vi 
i/-»*a»©«i»©*^tcffia-r*««K»i"jr*» 1 © 

^j^fiB tffiO/V7VB«K *t^-f 5 S^ffiB t © H 
so ©, K®iStlli:®iSSOiM4)St5S30Xf'yy 
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[0 0 2 0] CCT. ±e+M>V9vMMt£liW* 

[0021] cntcto, ^ttfrKffls-rsfBiaoiBtfi 
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fgTT % k 3 m&*ffi <r c k #t* t So 

[0 0 2 2] 4*5. #«UL;fc««©#*K:-OVvt, KW 

*k*©tt©**k*EBU"*-s K«wicia« 

[0 0 2 3] *RW©Bi«£rittfiite:&^T, ± 

E«»M«*e*#to-e:T«ffca:/ V *7 vjli^O-gp^ 

mmft<Dmm<D¥-t£3mmm*Mib%wi i ©x^-y^k. 

±ffig|5#®$!<D&*lcOV>T. Kffi#ififil|*I<D&iSX«) 
©Xt- >y :/T-*#fc¥i$eJf JKffiO^fi t-*T S J: 3 

&{itcif je-t § ms 2 ©x t- ? y t & «t -5 tc l tc 

o¥iSW*«**ai>*JB i ©x-r-y :/k. m^o 20© 
^®#CD^©^fCOV^, c*ie>©2o©g|5#ili«! 

©*»»K:tiM*-siH*<0iji!ft«*, ±§esi 1 ©x^>y 

^T*i6fc¥i^Witfil©f IIBfflKWiE'r 2 OXr7 

wmwmm t ±nzm 2 ©x^>y r t»« m u fcnowai 

fit k OM©reilfC*S UT, _hf2lg 3 ©X T" y -/T'&tb tc 

m#©ks# >>4 < 4 s £ 3 4ffi£*46. #«>fcffi*E 
a* anirr s% 4cDxxyyfc^rwrs«t3 

[0024] cnfcttK «»©as#iE«*ffl#^to-& 
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[?gBJl©Iflj8!<D7£<S] fcTF. ^RE^fiS©^^^ 

[ 0 0 2 6 ] m 3 «*«l!l!©ia«£jaWS;**« bfc n 
>tTa-^^X^A(DS*W4/N-H^xT^^-r 



[0 0 2 7] 3 0 OttH&^gS, 3 1 Oti^r 

<^X. 3 2 0liX*-v?-. 3 3 0«A^7, 3 4 0li7* 
o-b-y+h. 3 5 OlifStigfi. 3 6 0 (iHJJS^XT-fc 

[0 0 2 8] 03fc*3^T. 7J^-7 3 3 0(i, 

*-vt3 2 0«#lkiH<$£IXt>&tyfcto©fc©T&5o 
[00 2 9] 4*5. *^73 3 OVX^-v^-3 2 0iC& 

«, BiKtU:. 15tggB3 5 OfcfeS^toStf. 

T^o^t 1 — *T-a&3*§"&{i. @7KLTi^4v>A/Dg 

^B3 5 OtcKffll^toSo 
[0 0 3 0] Sfc. 03K*5l^T. il^!«^SH3 0 0 
«. ts^=7 3 3 0-^X^+^-3 2 0tCj;-=>TifcD&3;tl 

3 fc 46 © fc OT* ifc 5 „ 
[0 0 3 1] 4*3. «r3<fc3tc. J&tf^ti. BHR£ 

3&gtf;fc3©T-. B^SE3 0 Ofcli. #fgBJ3cD 
fiSt^tv^iii^fea^^ ns c k i:45„ 

[0 0 3 2] Sfc, 03^C*3V^T, V^X3 10{iS# 

> tf a - * ->xfi» k j»f^# t <df,b ®« fsea^ i: 4 

So 
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[0036] te«^B 3 5 o±(Dmm.mm 400 ic«, 

40 Aftlfeatttf«M2ft«ttM1M(4 0 5 k. cto^ip 
XLfciB«SS«<«lfiSn*tt«iHi«4 1 0~4 4 5 k. 

[0 0 3 7] 4*3. S»tt««fc«ttSft*rtSK:oV'> 
T«. *«Wfc^S@i««iS©ittWkfttc^5o 

[0038] 01 «*»w©®«^ia*ffi**«-rsfc 
to oia^saii© jaa«s*sf 7 □ — =j- -v — h t-& & . 
[0 0 3 9] ST. #fgBj§(c<l5iB0aag{c*5Vv n i. 

Xf77"l 0 0T. H^A73-ri.o 
so [00 4 0]i¥L<«. Xfy^lOOTli, G«9EB 
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3 5 0±<DKtt««4 0 0im&-£tircteMffiM4 0 5 

(H4«»ssnftv\ ) xJiLtmrn (A*® 
[0041] ccr\ xtimmtiz. f-f-^^w^7 

* * 5 OHP<D(H]K$ft t tf-»f * «fc 5 K LTit^tf ft 

snfc f lt, xf7^i oom in— 

fiIT*f^y?;W^70i=if *— SO SHEA ft cT £ fc 

So 

[0 0 4 2] $fc, 7fv^l0 0?tt, ISOA^?: 
^SffiiT^^IiLTS! t> &AJ£ 2 ffi©JH®$r A2rT 
[0 0 4 3] StV^T. 0 5T\ Xf'^l 0 

^oTv»«j:3ftj£ftWfciiHi (/V77it) 

[0 0 4 4] iL<lt Xf-y^l 0 5T?S, 0 
■ 5 fcjjVTJ: 5 fc, (5 3 0) <DP3f§[*-7*;l/ 5 0 0 

*MEJfcU ioB©A*H«tt?*S rfli (5 1 

o) j ^cor^^s o o±fcsub, raia-tT*;^ 

5 0 0©*ffilC rfll (5 1 0) J «rftS*)Wt*o C 

[0 0 4 5] 

[»1] 

x' = d • sin(atan(x/d)) 

y* = y • x'/x = y/x * (d • ein(atan(x/d) ) ) 

• - - (Si ) 

[0 0 4 6] CCT\ x, ytt N r?Xl (5 1 0) J 
©¥BS±©&«rz?&?K x* , y' RjfiF&-r;l/ 5 0 

H\ RfS*-r;l/ 5 0 0 OI-dO (RlSS^e-r ;U 5 0 0 ©* 
i*5 4 0O*ft) r?Il (5 1 0) J ©are 

[0 0 4 7] *LT> 2oB©A*W*T&S r?l2 

(520) j (covTts rfn (5 i o) j tvm 

<Dfi»%ffV\ RWFe-rJl/ 5 0 0 ©«BK r?X2 (5 
2 0) J *W»> raW-So COfc** riR2 (5 2 

o) j £ rfii (5 i o) j x-fS?*/!/**^ 

Oft**-£©IIrtEftft£Wft*T»i&LT8l9fcA,JS 
HHTPfctK HU»fe#^oTV»SSS#tfftft*J:$K: 
LT, r^H2 (5 2 0) J ©ftSDraWSrffSo 
[0 0 4 8] ^CT% tVS>*;1/**7©II«EWi«:, P3 
fti^r-r;l/ 5 0 0 ©efiDtt 5 4 0 tC— SW £ * r> f 
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tffctK A/* * 5 ©mfift ft ii> 

KWICjRfe*^«J:5ftftft4:-r*<M8«<a&*)» Sfc, 
RHf*7*;l'5 0 0©¥&d (5 3 0) tt, (Wtf/l/S 
0 0©«ffilcifi?>raW&tt3 1 OOH«*i*trtlft*^ 
xV S?*;V*^90*¥^FlRiOHft fc— Srr* J: 3 ftfii 

[0 0 4 9] 30@©A7Jlii^, 40g©A73 

Mfc, P31B*f*;l/5 0 OO^ffifcRSOraW 
TV^< C £T% 1 ffl©A*H^5. 1 
10 *£i#r* C 

[0 0 5 0] ft}3, ^/^v^K^r»Sfc46©»ffl^B 

7sf"y?\ 0 5*#< Cfc*<T?*So 
[0 0 5 1] /V5"7M*, 
s?* 5 ic <fc S JUBffill* >frC» i: b fc l jgtfoftfe 

i^fci:fcofc/Jv£vMKHTM&oTt>eb3£*.ftv\, 
[0 0 5 2] c©«k3^LT, xf'yyiosm '> 

ft < tt 2 0©RftStMte:j3W*/V7"VlB»&#* 
20 4fi!cU I21«gB 3 5 0 ±©IB1t«lg 4 0 0 (Cfit{SS£ tl 

fc*a»w«4 1 o (H4*#sB*nfcv\, ) ^figb 

[0 0 5 3] igcV>T, Xf7^1 1 0T\ 0 

5T-£$Lfc£?&©/V7V®#©&*fc:o^T, K/< 
/ ^©*fe©^tc^oTV^^{*lCfSC 

t, «is©ffi«©®^ cwmiio 

[0 0 5 4] i¥L<«\ Xf->7*1 1 Oli, fi&J*.t£, ^ 
if3-2 1 8 5 8 l#£«fc:eft© HlHii-te^^T- 

30 »*spjfflu JtfNi3^*x3 i ofr&A*rrs»wt= 

5. 

[0 0 5 5] Xf»n 1 

teOl^Ts KWi^H^rt©^!®^ 

gi3 5 o±©f5it®«4 o oim&-£t\rz&mmm4 

40 [0 0 5 6] «!©TX^iBI*ti, 06©6 1 0 

iCTK-t&olc, *<D&wmi)\ /V57i«6 0 0O« 

HjRo-r-^afciaiiUT*-** s^h t 

S7-?m&6 0 0(Cfctt5^®^©7^-^ (RGBf- 
^f) ©ftto tx^I^t- ^TfeS<}:5(c-r 

[0 0 5 7] 09* WT, 06(c^-r.J:^{c, /V 

so jALft:/V77ffitttT%«it&tt. K76 3 0AW7T 
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v*ffli*B*6 5 oicftfoirz-vT.t'mtLT r l j % 

&6 3 Stf^oTt^J&WWBffce 5 5 
•7X^{ii:LT T2J IEffi©M6 4 0tf¥^T 

C»*flWBB«6 6 OlcttJSf &v;*^fiii;LT r 3 J £r 

SUStlllfee 0 5A^oTV^ffii^ffi#6 6 5fC*f 
JS-r^vX^<li;LT r 4 J fi*?TfefOi 
6 1 5*Wfr3TVS««H«l6 7 0 tC*fj£T SYXi'i 
i: LT T5J fc^S., |ta*>oT;fe¥©g6 9 Oi^oT 
V»S«iaH«6 4 5lC*tlfo-f S^X^ffiit LT T6J & 

*©fl&©«WE»6 8 5lC*fjS-f 3V7.*<I£:L 10 
T T7J %-^^.5o 

[0 0 5 8] ^VT\ Xf7 7l 1 5T% 1 

[0059] /V5vH«#3fc^LfcH«;fr&*bssft 
s^s-rsidJc-rs,, 20 

[0 0 6 0] •?• LT. 0 6lC^-f iMc, 7f 

y^l 1 OT'-^^fcvX^fBCTttc, K^X^flSr-S-x. 
fc*«iB«fc»LT£*Lfc*tt1ll«6 8 0%*ff5tttt 
Z>rc*b<Dm&7—-7;\s6 2 OfcffcSU 1E«^M3 5 0 
±©821t«« 4 0 0 fcKffiSftfcffifMiitt 4 2 0 (0 4 

[0 0 6 1] &:}3. 0 6tC^Lfcfi«jT'{i, ^S^CTi: 

0£*5V>t, j«Bffl[|i*»6©E«l«'Bli:l»f*fc:»*srs 30 

[0 0 6 2] CCTte, ««©##J:fc it/Ml** 

© H?t frit Wfe fflg-r set rThI* A^7f$S<l;5 

u = (X.P)-(Q-P)/(||Q.P|ir2 



[0 0 6 3]$^T, Xf77"l 2 O^iiyx-r-v^l 
2 5T% XfyT'l 1 0-e#»JLfctt»©««iB«fl!>«- 

[0 0 6 4] C©£S-&#©JElfc&ffifcLTt3\ 
If, TFeature-Based Image Metamorphosis J tCxE^P) 

©fi*«*jww*. 

[0 0 6 5] JT, SL< «\ Xf77l 2 0:fc<fct/X 

•r r;V7vii 1 (700) j *5<tt>* r;V7vii 
2 (7 1 0) j ettv-r* »#7 20^7 40, 1 

^7 2 5 kMft7 4 5. »»7 3 0 £*g#7 5 0*» £• 
*tt£Wtt*«#ibTift&$-*i:. 8BttSB3 5 0 ± 

©E«ai*Ka««tifc«M«iH* 430 (04 *#jsas 

[0 0 6 6] &*3. C<DtS«ft«ti4 3 0{C«, 2£>fc:, 

[0 0 6 7] 8S^T, r^y^viB^i (700) j i3 
<kt>* ttv^mfcz a 1 o) j ©=&>«? fcoi^T, k 

R/V9^Hiflfcrt©£iH*©S*«T?&SgW'* 
* Wl/Dfc, 0»2) fc«koT*ii>, SB1SSB3 5 0± 
©ISttflS* 4 0 0 Kfil&gftfettlAWM 4 2 5 (0 4 * 

[0 0 6 8] c©«Wr<* b;l/Dtt. UPgUcti, H«S 
IBloiSiilcioTaisn, c©<fc3icLT&IIiiStl;: 

[0 0 6 9] 
QR2] 



v = CX-P) • Perpendicular(Q-P)/( I IQ-PI I) 

X"=P'+u - (Q'-P)+v • Perpendicular(Q'-P*)/(| |Q'-F| I) 

• • • (»2) 



[00 70]fcf£b, P, Q«x r;V77i|l (7 
0 0) J K*V>TJg£*nfc»»7 2 0©(S4«*»j5fc 

fS^^h/K P* , Q' it. f't/^-vmm (7 1 
o) j te*vTigjEsnfc*#7 4 o©pf**»jfifc-r 



so 



r;V5vi*i (7 0 0) j rt©ffi*©ffiH**j£i: 
X' ti, X«»ftLfc«©ffiH*»j«k 
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[0 0 7 1] Perpendicular ( ( )ft<D^ 

CO 0 7 2] fcfc, »j£«nfc»##«»#fcS»£ 

fefc^fK* h,l/D i «HM)Ki«V^^* 
<*»K SIS £'h2?<&5 J: 

9&M2*W i **Cfc«W i D i CDfD^ W i ©fPTRfc 
LfcB*\ teWj^thAsDhtZilolCtZo £(DW.fr 

CO 0 7 3] 
C&3] 

D=<S Widi)/(2 Wi) 

• • • (»3) 

CO 0 7 4] JfcD&ttT***, (3&4) tC<fct>«£ 

5W i 5:ffll,>T^5 0 
CO 0 7 5] 
C&4] 

Fi=flength A p)/(a+dist) 
Wi=Fi A b 

• • • (»4) 

CO 0 7 6] fcfcU length^ dist 

H^ifT'OES, a, b, ptt % ^T'^S 0 
CO 0 7 7] c<D<k-Mc, 5*<£>»*R±«-&TiH«*: 

CO 0 7 8] ftfc, Sg*^<0^**j*Kt4, iS0±{c 
t> n^Wigi1*}<D'£T<D& (MM) li. If±0 4O<D 

CO 0 7 9] I^T, Xf^yno-i?, "rj'JZfrio 
* =7 K A 5 »»ffiM«rSm £ L ft * ©«lgS# G*fMlf 

^sa^risi^A^-rSo 

Co o 8 o] »L<iis xt'^/i 3 om mz.i£. 
tc«^?n«:%0±(D^m^}t^-r 5 «t b tc-r s c t 

ft§U iBfl&CT&B 3 0 0 ic x 7 * /!✓ h ttMti: fctt 
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CO 0 8 1] m^X. Xf 7^1 3 5T\ Xfyyi 2 

Lx &#Lfc4MSttH«* % Xf77"l 1 5T^8Lfc 

CO 0 8 2] £35, 3 5<Di¥fflfc:o^T«, 

10 0 2tc^-T7n— h^v>T^-r§ 0 

CO 0 8 3] Wifely Xf77l 4 0T% 7.^-y^l 3 

CO 0 8 4] i2(iXf-yyi 3 5 ©KUBfcttS**-*" 

CO 0 8 5] 0 2 IC^cT «fc 3 Xf7y2 0 0 (01 
0X77 7"] 3 OlctB^LTV^o ) T'Wu 
# *7 K J: 3 i: L t # O+B *J 

20 ^^-r^^(^%A^-r§o 

CO 0 8 6] fe^5A/, »f£&B;&S*1&i:Lfc£:£<Dfa 

t ut, 2 ooaa^«*«i*«»±-e<Drt»it*flsi 

9, Yi^tLt, »»{5iSH©IEf8t% r i j tLtc 

co 0 8 7] &it. mm-xmic-o^xii, iBSJt©» 

so ffi*A*jLTfeJ:v>tf* srt©»£T**Si:, 

X i: v ^ A2j ©ft* pflJi-e* So 
CO 0 8 8] St, V>S, 0 1 OXf7 7'l 0 5tC<fct? 
T, t& 2 0©/V^il«i#£fi!t£4Vr^.g> 0 

SP>tC. 0 1 0X7-7^1 2 5tC«toT, 

CO 0 8 9] ^Cf, Xf7 7l 3 5T*«\ *-f, Xf 

40 7 7'2 0 5T% 2-D<D^y^mmti^Amp>t\rc2-D 

CO 0 9 0] CO*IHHftB©«€(4, ^ibmnbtc 

so co o 9 i ] mz.is. m 8 fc^-r<t 3 k, ^ttss 0 
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^fflB8 1 OicfctT^/V^vB^S: r/V^iSfg! 
2J i:-T5i:t, 2OC0jf^ftB8 0 0. 8 1 0^ 
*§17t8 3 0±cO<fH{ifi8 2 Otf&SSftfcii-rSi:, 
^mW.m8 2 OK&tt^Ra'V^vBS! (cfHtttt8 
2 0{C7V^*;l/#y^tffc3^K*#£n?>T*&3 7 
^y^B®T-&3o ) tt, r;V5Titij6J:tf 

[0092] $fc, m%.if. 09jc*3v>t, /V-^tb 

#7 0 0«ffiI8 0 Ote*»* r;V57itlj 
1 0^S»{itB8 1 OlCfettS 
r/V57it2j T-feD, /V7VB§9 0 0WB 
ffiB8 2 0{c*5it§4>F^/^/^Tiiii^T*fe§o CCT- 
& SHe*ftfc4»Hffi«8 2 Oft\ ;V77l§7 0 0 
>t»OKT9 1 0<0{4Bi:^y^vi®^7 1 0*<OK7 9 
2 0<D{4St<DP^^^7>C>4>^Tfei.WI^LTfe 
5, WU'V^vMRO 0 0*<DK7 9 3 OcO{fifi«, 
/V^VilKf 7 0 0 <P<D KT 9 1 0 <D®M £>V ^B 
<f!7 1 04"£DKT9 2 OcOfsiBfcco^FiatC&Si'McL 

[0 0 9 3] m^T, Xf7^2 1 0X\ 0 
0 T*\ti L fcSjSffiBfc r-f ~>* * ;P * y 9 tffc S 

mzrLZT'&z ? ^w®t cr&fc-&, £jst^i? 

0 0lc«StSiI4, ^i6^46 fctSWft 
[0 0 9 4] cn«> 0 S'CifetLT'cft'RSffi 

«fc*w-**isi/V5vis**£»£u cn&®*M>< 
^E3 o o(c lmcmTjkT'ZzwmitmztiT^ziD 

[0 0 9 5] M^T. Xf77*2 1 5 T% £t\ X-r-y 
7"2 0 ST'^L^^F^B^fctTS^F^V^Viii^ 
*£l«U £/£Lfc4'lll!;V'7VB«<0&*{C'O^T, 

[0096] «t»iB/^y9-rwu*ftj«-r*fcfto 

[0097] sfc, tfF^v^-va^^fig-rs^ic 
tt, iio^f-y7'i i o-e#»JLfc««iii«*M»« 

7i i 5T^«Lfc«tti«atefleo?£iR»if«»'&j«-r 

ifiva ^/jxs i^ffionttflffli*ft£tt£ nr cot% 
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[0098] m%.i£. m i otc^-r-t-Mc (as 1 o o 

OlcfclT^V^vB&l 0 0 bS^IpJ^ 
(p] l^fc/V 5 -?mm 1 0 0 5^r»rfctc^fig-r?>fci6tc, 

mmaLm 1 o 1 otc*3it§^y^Tii#7 o o, 

t>\ JS^&fi 1 0 2 OlCfclT^V 9"Vlffl»7 1 00(5 
*HC t^F^feB 10 3 0-10 6 0&C*5tt£'f FI/V7 
■^Bilitf^JiScStiTV^ i:-T5 0 
[0099] c ct\ rfite i o o otctsv^TrS]^ i o 

10 7 0&cJI*.3*feti, {&fil 0 1 0tc*5lj5lRli:(p)^ 1 
Ci:T% fitB 1 o o OtCfctf^V ^vlHg! 1 0 0 5 co 

•5-5. 07 ofcffiMs^sawfroiHflwi, /<y^v 

B^7 0 0<O7^c0. 1 0 7 0 tcffl^-r ?>a57TCOiii 

««ie^«uffi bT^fig-r s c t * % t v> 7 c t tuft 
i o 7 oicmm-tz>&#<Dwmi)\ am 1 o o oiasv 

20 [o i o o] mmc ^(om^w^mmi o 2 ots 

.fcWF^y^vBftl 0 3 0—1 0 6 OfCOV^Tf^ 
filBl 0 0 OKfett-S/V^B^l 0 0 5^\co 

fiLB 1 0 0 0&£*5&7£/Sy5TiIi$! 1 0 0 5 ^rjg'fW^Jlc 

[0101] i^awtcti, ipj^^-gt-r^^y^-riii^ 

±cOfi[fili 1 ocO*.T-$>^.^f3, Stc^-S'^y^^iii^ 
^ ^3Sfc tti&g te ^ ^ *^ as i. SAiOtl^r tofc® 

[0 10 2] ^CX\ Xf77'2 1 5T'^J6?.ia^!(i> 

2>£5lC-?%£ttfX*t. m.<^7-y/Z 2 0T% XT" 
7 72 1 5-t?4BgLft(fH/V7VliOMtCOV> 
T, 77 7^2 1 5T*»fc«fl»tt««iii«*«I5Bl 
0> IB1SSB3 5 0±cof31tKRJ§?4 0 OlcmUZtxtcfe 
40 lfll«4 3 5 (H4*#BRStlfcV 0 ) WDBlofc 

[0 10 3] 443, COfflffittW«iB«». ^K:, 3S«I 
^fg^cote^S^Sfi^ ti% o 

[0 10 4] WAfef, 01 1 lCvHr£5iC s 2-0<DWim 
ffiBcOF^coaBlIt^ 2 L t L> ^c0H(cl?W|ig(c n 
Hfi[B?r^5«^{±, m-&7{afil040. 1050 
F^c0iEgt«, 1 = 2 L/ (n + 1 ) t*^n§ 0 

[0 10 5] ilc0J;7{cLT. Sggt 1 C'fclC^F^^y^ 
^B«^ffifi?tiTv^i:{S^-r5i:#, fitiooo 
so iC&ttS/V 77it 1 00 5^ ^fl&«®J^ 112 0 
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~ 1 1 5 0©<fcMc#9JU *Jfifl»tt««OBi««. Vi 
SJf, 1 2 OtcO^Tli, (Sfil 0 1 OK 

*3^§^y7vii^7 o o*>p>^]t)^ofc^mt»ctRi^ii 
^e»^L, mmwmwi i 3okov">t«:, *fta{s 

El 0 3 0£fctt5tfM/V7VH«frCWD8tofc$I 

wtt««ia«fr&£j£u fflw«««i i 4oko<^t 

«4, (PNffiB 1 0 4 OtcfcttS^WV 7-7l«A>5« 

o«i-Dfc)afl»tt««is«*»5*iSL, mmvimmi 1 5 

O^oi^Tti, 4"Rg(SB 1 0 5 OlcistfZfpFS/^/^-r 
W®fr £> W K> m. o fc&fflttffiitfBttkftt 6 5 C £ ft 

[0 l 0 6] ±iBLfc.£-5K:, Xf-y^Z 2 0&c<ko 
So ) ££fi£U IB1SSB3 5 0±©IS««^4 0 OtC 

5t«^tifcts*ft®^4 4 o m4t£»mztirz\,\ ) 

*A!lJI*flM-J:$fc-*-S. 
[0107] t/^. 0 1 2 (CaVT j: 3 IC, (SB 1220 

tcfcHtstffc^^y 7-7®^ i o o 5%^j*-rsfc«) 

fc, (SB 1 2 3 0(C*3ttS/^/^vjii^7 0 o^e.^io 

fljiofcjaflB«fi«iB«*<, mmviffimm®i 1 20?$ 

t>, (SB 1 2 4 Olc&ttS/V^vHB 1 0 5 0fr£>gj 
OffiofcfifflttflWiiiBtf, fiffi^WMiiOflti 15 0T* 

[0 1 0 8] ;V5va«i 0 5 0tt, frfc^V^T 
Btl 00 5^ttf, (SB 1 2 2 0 l£l3lif %>W(8J5 

iftt— a-rsA^a^ ^y^T®#T^0s «ot, 3 

O<D(Sfil 2 2 0, 1 2 3 0, 1 2 4 0 fttC> (SB 

l 2 4 0^illlHft^<DiSftll^i:-rS3o<D]l^tc^ 
ttte%t>\ CCT% (SB 1 2 3 0 i:(SB 12 4 0 

t<Dm<Dmm& l i t*& o „ (sb i 2 4 0 t(SB 1 2 2 
[0109] set, mmwmwmmi 1 5 o*vv5 

viS^i 0 5 0©— a?^rS(gtcffi±Lfcfe<DT'fe?)i: 
KflftttBigBBl 1 2 0ti. ^y^^rmB7 0 0© 

-»* (&5) of5*s* T'te^-r^idfc-rtitf <t 

[0 110] 
[&5] 

S'=S • V r "(Ll*Ll+L2*L2)/L2 
■ ■ • (#5) 

[0 1 1 1] CCT% S<D(i«, mtE<Dm.&tiLWfrt>Z. 
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[0 1 1 2] BBtC. 7*7- V 3 5T\ Stfc&'V^ 

via«tc GRD ©aBe»**u e^-^ffitcs^-r 
hsb 3 5 o±©ieii8ii£4 o oictt&-& nrc&mm& 

4 4 5 (0 4*#S8£tXfcl,\> ) fC, KlfcBB 

[0 113] ^p.{c, c<Di3KiBfii*ia«i«^B3 0 

[oi 14] vLtwrnLtzZoic, xmivnummic 

[0 115] COPSHC* *E100!W«ft* 

¥ot*D, *ft6©ffiBM«tfiH»t=J:oTAn»fe 

20 [0116] sfe, *fzw<Dmtmmiz&tiif, g&s 
!M^BKfctt«>v^v««#«ffi2o;fcntf, -7* 

[0 1 17]SfoT, ^LfcMB*'*— XfCbfcTV 
-tf>T— > 3 v^X^A^©^— ^-wl/'J TUf-f ->X 

(#*^^;*-*©fWJgB^fS&B30 MSi 
■e-fc, ^*-*x;b-t§Hc&tt5^i!S^HS:#lSiU 
30 fe**1^{MT©B«*«BTSCfc*^fJii:fc*. 
[0 1 18] JCU:KBWILfc*««»K33V<>T\ 

3 5 otc^LT*3<<k^tc-rtxif, #mvM> m<o* 
[01 19] &it. yjcicmmLrcnffiBmicts^T 

[0 12 0] (KDJc^tc-r^^fcii, 02OXfy7° 
2 1 5T*, 0 2cD7.^-yy2 0 5 T'^SLfc'f'F^ttfilc 



(11) 



WM¥ 10-208074 



19 

[0 12 1] ^LT, 02©Xf- t y7 p 2 2 0T% 12© 
T, 0 2cDXT--y^2 0 5T'j*SLfc(fF^flLBtC*5lt§ 

[0 1 2 2] &33, ®TO«®^iii#%^fi)t1-i.|^{C«, 
02(DXT772 1 5T*SS3LfctH^il^. 02©^ 

co 1 2 3] seac &.±icmmhtzmmmmit. a* 

CO 1 2 4] tt*rtf*-fX;P-: fX^-MC 
tM»©/V9Tiil«*±)aLT43<o *LT, 

CO 1 2 5] cc*r©jas**ffrsci:T?. tt^rt 
co i 2 6] cnfc^tK *wre£j*snT^«/v 

sfetR c«fe^«^-r zct tf^sm ttez, 
co 1 27] tcz?. &L±.icmmvtzmmBmic3s^ 
Tit. ^etcx &cFic&^%£?ic, mm&m&mm<D 

CO l 2 8] 01 na»©/V7Viii*i3J:tf4»IIB 

CO 1 2 9] 0 1 3E33l^T, 1 3 00. 1 3 0 5, 1 
3 1 0H m©/V^Hi»£fca:4>|ffl/V5 
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CO 1 3 0] -ISWfcli, «»©««©¥*9iW*«aic 
T*&tK «oT 4 01 3tc^-r«t3ic, &JSflB#^IS 

CO 1 3 1] fCT, B«S©l8Btf»&iMcR#Sfc 
*6E, SffiKIR«i9ttO&n«nB-r«A31«fT3J:^ 

CO 1 3 2] 0 1 3cD^*7 7lC^-rj;3lc. 

mmttmmmm 1300 rt©*®*©^ 3 ?^^ 1 
10 325 k> , BBtt&wsB 1305 ^©3- 

i«ig©MA¥^fii^, 1 3 3 o\zm?mT*$>z>, mmtt 
bbbb 1310 ft<D&mm<Dwm.¥i%mt}\ 1335 

sBSMBfcffiB-rsHBcBBB*, cnsofsw 

ffi©*HffitcfiiErsi:#K:, Stco 

lfite*SJ:5K % ^WgWtfifiK:fifi-rSBf£©WfSB5: 

Co 1 3 3] c©£?&»&fT5ci:T% 3Ef£L;fc/S 
20 /5vMB*«»W*WaH«fc*e«ftBJW5S£-r 

co 1 3 4] nit. &L±icmw^rz&m&mici3^-z 
it, ?f>(c, wTtciE^sjc^ic, fpm>W7-*wm* 

LV\, 

CO 1 3 5] 01 4 It, 2O0;V7VlSlfrP.«fH^ 

©&&s4tff£?«jra*mrMi&B-?fc«. 

CO 1 3 6] 2-POffii 1 4 1 5. 1 4 2 5T'# 

so LTIfc 0 &A,/£ £ft©BB*r> 5£ric Lfc 2 O©^ 

y^-vs^e., tpnmmi 4 2 otctses^p^/^y^ 
vH«*afeja"rs»*%#*So mar, a 

IB*tei:&*«fe*E#fiELTV"»««M*l 4 3 011 (ft 
■ 14 1 5A^{i^ %lf*l 4 1 0&cH,tiTJI;L&^U 
fill 4 2 52f>e>H %l*l 4 0 5E^SnTm^.^:V^ 
-r&£*K if%5)©{jj[Bl 4 1 5, 1 4 2 5T-IKU 

ti, 2 0©/^y^Tjffi#^:MP^LT^St)^(cliV^^: 

v^©t% ^©as^ii^agpt^So 

40 CO 1 3 7] *-CT\ C©«fc9^aiT^US^»gi3% 
CO 1 3 8] Xf7 7l 4 4 0T% ^»gP%^m 

micmtimz&gnrztzit, zvmwwv 

CO 1 3 9] m^X, 7,t-v?\ 4 4 5T% ^«SB/1H 
so KffiB"r*liiSR*BK<» Xr-y^l 4 5 0T% Xf-^ 
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[oi4 0] m^r, 7>t-v7i 4 5 5X\ mmoiam 

[0141] £<D£?%:>>m*'i : 7 : ?££T:\ fH/V7 

s i: tee c 5^affl*«a-r sct^ 

[0 14 2] f fc, OJtKSiWLfcsfeaBgJIRfcfe^T 

■rsjaa^satff 5 c fc# Rjt£fc*5o 
[o 1 4 3] si 5«\ /v^-vaMM^ffitciaR-rs 
«yi*stafcfT 3 fc 46 tc « v > p, n s ^- 7;i/*:^-r mm 

[0 1 4 4] 0 1 5tc^-r*5ti:, C CD 7" — ~7 )\> 1 5 0 

on, mz'&vmm (ii«a^«3ooic«^n5 

5f-?^ a»M©/V7Viii«i 5 i 0©2<E>iBJR 
3^6tt*m-&tf«fcV>ft»fc^5B*(tt« (x, y) 152 

[0 14 5] C<D7—y)V 1 5 0 O^fflV^CtT, & 
fiWWfcTS C fc* Lie. ¥BB'\©]«^*SKA^«IJC 

[0 14 6] £fc, JUUiteBiWbfc^aSJglRtfe^T 

/^s 7~?m&%%.t$.-? z> tabic nm^^5 o o 

[0 14 7] 0 1 6tt^K*B&*7*;l/lcja»T«IB©» 
^»*5*TBW!BT&tU 1 o<01H* (a?*) 160 
0** J*0>f-E»l 6 2 OlClpJltT, J*S® 1 6 1 01C& 

[0 14 8] 0 5lC^;L;rcP3flPET\>l/5 0 0*ffl^5« 
t-T is* foil t^CDmWtl*, 5 0 

oo*-Mi5 4 oic— st-r^J;3icL*ttntf*p.*A^ 
[0 14 9] co&SK. ±»Lfc/V9'7llH*©ttfe 
[0 15 0] *fc, WJLlcfWBLfcjtaBgaUcfeVT 

*rkc» auetcioT^iSKwetifcfMgjifcifiHi/v 
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[0 15 1] £j£&©#?fcfc/V7ViS«©*U:«\ * 

10 rshc, a»fcj:oT»&nfc&Kfiii:«fiai/V9vBiflfc 

[0153] II 7 fcjrrf «fc?fc, ;V7?Itl 7 0 

t— 7;H 7 1 0*»JSWtT43 
<c il<Df-7;l/l 7 1 Ott, fes&filfc&s*, «. » 
Ofe<Dfi!t^ffiR, G, Bfc. JHg*07*4 3'*/l'Jb;<? 
(D^^O^ffil (x, y) fc, 2*7£¥ffi±Tlft#0 

file fcjWEasnfct»oT?a&*o 
[0154] *5-rtiar, *ffcfc/v^vjii«*£ja-r 

%R»c. C(Dx, y, 0©fil^P)*fjS-r5®m*^S» 
[0 15 5] &*5, t— 7 1 Olt BfBf*-r\rt/5 

*fflvs»&of*— *«ja«» t^— r>n 7 2 0tC^-T 

[0 15 6] g:fc, W±teKWbfc*M0BttfcfeV»T 

so fffc^v^vB^^-r^jc^fc-rscfc^T^ 

-So 

[0 1 5 7] 0 1 8ti, ^»«*SnTV»*tB»©/<y 

[0 1 5 8] 0 1 8KH3V>T\ 1 80 0IJ, 

(x. y) *#*SW»**fca60r— r>WT*!), c 
OiftEI±<&S« (x, y) tett, aft<0ftfJ*Hfe 
40 K»fJSnft/V5vB«l 8 1 0tf**JSW*&tl-CV 

[0 15 91 COvV5vi«18 108, r«Kl 
(1 8 20) J~ TfBJgU (1 8 4 0) J OJ:5(C, f 
a6*B»©1H*lii«te»t!IS*iT*0, 4HMMKlctt, 
^P)fC, «14x-^ 1 8 5 0-1 8 8 o*^j5SM»e>n 

[0 16 0] ^J^if, l (1 8 2 0) J iClt. * 
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JSttx-* 1 8 5 0- 1 8 8 OtfttJ&ftt* 

[0 16 1] ctxtcfei?, ^fig-r^^^Tfc^/Vvviffi 

So 

[oi6 2] &*5, ^#>mmirz>m®L<D^s 7vit 
£t jcmuucaae l tux *> i&A,*ssi«*» & l *t 

[0 16 3] £e&C, W\ XU7h*^7 

[0 16 4] 

» P> ttS T* 5 3 ® «fc#r fc tC £J0W %> C £ * S o 
[0 1 6 5] ££fC % C<DBKC N ^{C noiWt*ft« 

[0166] *»wicj:n«, i^aa^fiit 

-f>*5*^ :/fcJ»fNc£fc>-&T»RT?* S «fc 3 

So 

[0 16 7] S£oT, ^LfcHfR*'*— X*CLfc:/U 
If >T— >3 l">XfAf ©/^-f t/l/ij 7'Jf-f ~>X 

TOH^^aiS-r 5 c i: tfpJAI i: * So 
[01] *»ijli©iB«£»a^**Si*3fc*©iIi«81 
[0 2] SlOXf'^l 3 5 ©ffiMfflfcfflS**-*- am 

0. 

[03] *»W©iB«*J«^rffi*ll«Lfc3^lfa-* 
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0o 

[04] iEmmmtmmmmcmuztiztewm&z 

[0 5] iS<Dli^e,/V^viii^%^-r?,Ki(DUi 

B£0 O 

[0 6] ■vxi>m»i5£xfm&<r--?>v<Dmm*7FL?m 

[0 7] 2 0©/V^vH&;&^#;£ffl^TttJSttttS 

10 |S<O3ttM0o 

[0 8] 4>M>V5vffi«*fls;5IB©BMiBI 0 
[0 9] «f«/</9vH«©ttlt*^"rWWH. 

[01 o] ji»tt«*«w;ai»±icja:vMttB'p»6n* 

[011] /V^vil«*5j;t>*4'^^y^'x'®^i:, c 

mmm 0 

[01 2] ;V5vafl*itf[|'|B/V7Vlfll4»6, 

20 T^-ruiH^0o 

[0 1 3] «ft©/<y5vH*:e*tf*|IB/V^HiB 

[0i4] z-zxD^y^mm^^m^y^m^ 

[0 15] 'V?7Hft«¥Blc£a^Sjra%ffij&K: 
[0 16] ¥K*S**7*;l/»cfi»**l«©8M^»*3S 

30 -ri«B^0o 

[01 7] gffc&/<y9vH«*£ia-rsi8K:, »f5&c 

J:oT»6tifcft«Mi:*IIB/V9TH«l©ftje«fc* 
[0 1 8] WR0W»ffiBlc«it«/^? < 7Hfl^&, 

3 0 0-M*S8i, 3 1 0- T^X, 3 2 0-X^F- 
tt, 3 3 0-*^7, 3 4 3 50- 

40 ffilS^B. 3 6 O-ffij&r^ 4 0 0-Eim, 4 0 

5-\iim»mnMVu 4 1 o-/V7vi«fttti 
*, 4 1 5"-vx*ffi«w&tt««, 4 2 0 

JHMflflWL 4 2 5 -»«r<* h;Mft»««, 4 3 0- 
»f&n««Mffii«. 4 3 5 HSffltt«i^«l^«W, 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may 

not reflect the original precisely. 

2 **** shows the word which can not be translated. 

3. In the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim l] An image with which a wide angle -scene centering on this view 
location is reflected about each of at least two different view locations (a 
"panorama image" is called hereafter.) An image of a portion which looked at 
this sense and the same direction about each of sense seen in a desired view 
location (A "partial image" is called hereafter.) An image generation method 
characterized by having processing which generates the new panorama 
image which will be generated from an image which cut out from a 
panorama image containing this partial image, and was photoed and 
captured in a view location of this request combining a cut-off partial image. 
[Claim 2] An image generation method characterized by providing the 
following. Processing which generates an image (a "panorama image" is 
called hereafter.) which projected in order two or more images which 
changed, photoed and incorporated sense of a camera for every fixed angle of 
rotation about each of at least two different view locations in this view 
location on the surface of a cylinder model as doubled a lap portion of these 
images mutually The processing which generates the new panorama image 
which will be generated from an image which cut out from a panorama 
image containing this partial image an image (a "partial image" is called 
hereafter.) of a portion which looked at this sense and the same direction, 
and photoed and captured it about each of sense seen in a desired view 
location in a view location of this request combining a cut-off partial image 
[Claim 3] An image generation method characterized by providing the 
following. An image with which a wide angle-scene centering on this view 
location is reflected about each of at least two different view locations (a 
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"panorama image" is called hereafter.) Processing which generates the new 
panorama image (a "middle panorama image" is called hereafter.) with 
which a wide angle-scene probably centering on a view location assumed 
from an acquired panorama image on a segment which connects a view 
location corresponding to each panorama image will be reflected Processing 
which sets up a view location assumed out of the above-mentioned segment 
Processing which cuts out an image (a "partial image" is called hereafter.) of 
a portion which looked at sense and the same direction which were seen in a 
set-up view location about each of the above-mentioned panorama image and 
the above-mentioned middle panorama image in a view location 
corresponding to this panorama image or a middle panorama image from 
this panorama image or a middle panorama image Processing which 
generates the new panorama image with which a wide angle-scene probably 
centering on a view location assumed out of the above-mentioned segment 
will be reflected combining a cut-off partial image 

[Claim 4] An image generation method characterized by providing the 
following. Processing which generates an image (a "panorama image" is 
called hereafter.) which projected in order two or more images which 
changed, photoed and incorporated sense of a camera for every fixed angle of 
rotation about each of at least two different view locations in this view 
location on the surface of a cylinder model as doubled a lap portion of these 
images mutually Processing which generates the new panorama image (a 
"middle panorama image" is called hereafter.) which will be generated from 
an image photoed and captured from a generated panorama image in a view 
location assumed on a segment which connects a view location corresponding 
to each panorama image Processing which sets up a view location assumed 
out of the above-mentioned segment An image of a portion which looked at 
sense and the same direction which were seen in a set-up view location about 
each of the above-mentioned panorama image and the above-mentioned 
middle panorama image in a view location corresponding to this panorama 
image or a middle panorama image (A "partial image" is called hereafter.) 
Processing which generates the new panorama image which will be 
generated from processing cut out of this panorama image or a middle 
panorama image, and an image photoed and captured in a view location 
assumed out of the above-mentioned segment combining a cut-off partial 
image 



2/31 



Japanese Publication number : 10-208074 



[Claim 5] It is the image generation method which it is defined as an angle 
which sandwiches one image with which it is an image generation method 
according to claim 4, and a path of the above-mentioned cylinder model is 
projected on the surface of the above-mentioned cylinder model is in 
agreement with a horizontal field angle of the above-mentioned camera, and 
is characterized by what is defined as the body with angle of rotation of the 
above-mentioned camera same in a ****** image is reflected. 
[Claim 6] An image generation method according to claim 4 or 5 
characterized by providing the following Processing which generates the 
above-mentioned middle panorama image is the 1st step which divides this 
panorama image into a field equivalent to each of two or more bodies 
reflected in it about each of the above-mentioned panorama image. The 2nd 
step which defines attribute information which shows sequence of nearness 
of distance from a view location which corresponds for every field divided at 
the 1st step of the above The 3rd step which generates an image of the same 
field as this field of the middle panorama images which will be generated 
from an image photoed and captured in a view location assumed on a 
segment which connects between a view location which was divided at the 
1st step of the above, and which corresponds from an image of this field for 
every field, and view locations corresponding to other panorama images The 
4th step which carries out superposition composition in sequence that 
attribute information which the 2nd step of the above defined shows an 
image of a field generated at the 3rd step of the above, and generates the 
above-mentioned middle panorama image 

[Claim 7] Processing which is an image generation method according to 
claim 6, and generates the above-mentioned middle panorama image is an 
image generation method characterized by having further the 5th step which 
fills up a pixel value of a pixel which this suffered a loss using a pixel value of 
a pixel located in the perimeter when there is a pixel which suffered a loss in 
a middle panorama image generated at the 4th step of the above. 
[Claim 8] An image generation method according to claim 1, 2, 3, 4, 5, 6, or 7 
characterized by providing the following Processing which generates some 
new panorama images combining the above-mentioned partial image is the 
1st step which calculates an average lightness value of a pixel in this partial 
image about each of the above-mentioned partial image. The 2nd step which 
amends a pixel value of each pixel in this partial image about each of the 
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above-mentioned partial image to a value which is in agreement with a mean 
value of an average lightness value which an average lightness value of a 
pixel in this partial image calculated at the 1st step of the above 
[Claim 9] An image generation method according to claim 1, 2, 3, 4, 5, 6, or 7 
characterized by providing the following Processing which generates some 
new panorama images combining the above-mentioned partial image is the 
1st step which calculates an average lightness value of a pixel in this partial 
image about each of the above-mentioned partial image. The 2nd step which 
amends a lightness value of a pixel located in the boundary section of these 
two partial images about each of a group of a partial image of two ****** to a 
mean value of an average lightness value calculated at the 1st step of the 
above The 3rd step which asks for difference with a lightness value after 
amending at a lightness value and the 2nd step of the above of this pixel 
about each of a pixel in the two above-mentioned partial images The 4th step 
which calculates a value whose effect of difference for which it asked at the 
3rd step of the above about each of a pixel in the two above-mentioned 
partial images according to distance between pixels located in the 
above-mentioned boundary section decreases, and adds a calculated value to 
a pixel value of this pixel 

[Claim 10] An image with which a wide angle-scene centering on this view 
location is reflected about each of at least two different view locations (a 
"panorama image" is called hereafter.) A panorama image with which a wide 
angle-scene probably centering on this view location will be reflected is 
matched by making this coordinate value into a view location for every 
coordinate value on an imagination map. Furthermore, the 1st table which 
matched two or more fields which constitute this panorama image for every 
panorama images of these, An on-the-spot photo panorama image which 
should cut off an image of this field, and a becoming image for every field in a 
table of the above 1st, And if the 2nd table which matched a field which 
should cut out an image from this on-the-spot photo panorama image is 
memorized and a coordinate value of arbitration of the above-mentioned 
coordinate values is specified With reference to the 1st table of the above, 
and the 2nd table of the above, further about each of a field corresponding to 
a panorama image corresponding to this coordinate value An image 
generation method which cuts off an image of a field of an on-the-spot photo 
panorama image corresponding to this field, and is characterized by 



4/31 



Japanese Publication number : 10-208074 



generating a panorama image corresponding to a specified coordinate value 
by combining a cut-off image. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[The technical field to which invention belongs] This invention relates to the 
image generation method which starts virtual reality systems, such as a 
presentation system using the image a photograph of was taken on the spot, 
and digitizes and treats the static image and dynamic image which were 
photoed, and which were incorporated. 
[0002] 

[Description of the Prior Art] As conventional technology which has relation 
in this invention, the contents are reported to the following reference. 
[0003] (a) - "A Study on Image Editing Technology for Synthetic Sensation" 
ICAT'94, 63-70 pages, and University of Tokyo -■ the technology reported to 
this reference In order to realize view modification in walk-through of a 
virtual reality etc. The position sensor is attached in the video camera, and a 
motion of a video camera is measured by the position sensor, and he 
synchronizes the image photoed with the video camera, and the 
measurement value (location data) which the position sensor measured, and 
is trying to record on taking a photograph with a video camera, and 
coincidence, and the location data which digitizes the recorded image and 
corresponds the digitized image - ******** it is made like. 
[0004] Then, at the time of playback of an image, two or more images near 
the view of choice are searched, and after adding geometric conversion to 
these searched images, they are the contents of having realized view 
modification by combining and displaying and not performing inside rate 
interpolation. 

[0005] (-- b) "View Interpolation for Image Synthesis (1993)", Computer 
Graphics Proc, 279-288 pages, and Apple Computer Inc. — the technology 
reported to this reference In the case of view modification in walk-through of 
a virtual reality etc. The set of CG image which carried out the rendering in 
the systematic view location for the purpose of realizing on personal 
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computer level without special hardware, such as Z-buffer, 0 Or it is the 
method of creating a middle image using the distortion function of morphing 
currently prepared beforehand using the photoed image. 
[0006] "(c) Epipolar-Plane Image Analysis^An Approach to Determining 
Structure from Motion (1987)", International Journal of Computer Vision.l, 
7-55 pages, and Artificial Intelligence Center. - the technology reported to 
this reference If the image which arranged "epipolar line (field which cuts an 
image horizontally) in order of the camera station is made using the 
captured image which was photoed with the camera of a large number 
arranged so that an optical axis may become parallel and regular intervals, 
the point on three -dimension space will become a straight line on this image. 
If the property " is used and it asks for the locus of all points, they will be the 
contents that a middle image is generable from the camera image of a typical 
location. 
[0007] 

[Problem(s) to be Solved by the Invention] In the conventional technology 
mentioned above, there are the following troubles respectively. 
[0008] Since the conventional technology (a) records the location data of a 
video camera on photography and coincidence of an image, an image is 
combined based on location data and the image of false view modification is 
made, the special metering device for measuring locations of a video camera, 
such as a position sensor, is needed. 

[0009] Moreover, if the conventional technology (b) prepares neither many 
images which do not almost have depth perception, nor images which do not 
almost have change between images, it does not change the physical 
relationship of ******** and bodies, or it does not consider the phenomenon 
in which a body becomes invisible from time to time according to physical 
relationship. 

[0010] two or more images with which an objective location shifted somewhat 
are used for the conventional technology (b) in detail, that middle image 
must create, and many images which photoed this conventional technology 
(b) in the range in which the physical relationship of the bodies which exist 
in a sight does not collapse in order to apply to the image which took a 
photograph of a sight on the spot must prepare. That is, in order to prepare 
many images to which the view was changed little by little, a photograph 
must be taken from a multiaspect. 
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[001 1] The image photoed and captured is used for the conventional 
technology (c) with the camera of a large number arranged regularly. 
Moreover, like the conventional technology (a) Although it is not necessary to 
record the location data of a camera at the time of photography, so that a 
camera may always turn to the fixed direction It is necessary to adjust the 
location of a camera correctly and, and many images must be photoed, 
moving a camera regularly, and the special migration equipment for moving 
a camera regularly is needed. 

[0012] Then, in virtual reality systems, such as a presentation system which 
used as the base the image a photograph of was taken on the spot, without 
not needing special equipment or preparing many images, the purpose of this 
invention realizes view modification in walk-through etc., and is to offer the 
image generation method which makes it possible to observe the image in 
various view locations. 

[0013] That is, the image generation method which this invention offers 
makes it possible to newly generate the image which will be obtained when a 
photograph is taken in the assumed camera station using the image which 
photoed respectively and was captured in at least two different camera 
stations. 
[0014] 

[Means for Solving the Problem] In order to attain the above-mentioned 
purpose, this invention about each of at least two different view locations An 
image which projected in order two or more images which changed, photoed 
and incorporated sense of a camera for every fixed angle of rotation in this 
view location on the surface of a cylinder model as doubled a lap portion of 
these images mutually (a "panorama image" is called hereafter.) The new 
panorama image which will be generated from an image photoed and 
captured from processing to generate and a generated panorama image in a 
view location assumed on a segment which connects a view location 
corresponding to each panorama image (a "middle panorama image" is called 
hereafter.) Processing to generate and processing which sets up a view 
location assumed out of the above-mentioned segment, An image of a portion 
which looked at sense and the same direction which were seen in a set-up 
view location about each of the above-mentioned panorama image and the 
above-mentioned middle panorama image in a view location corresponding to 
this panorama image or a middle panorama image (a "partial image" is 
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called hereafter.) Processing cut out of this panorama image or a middle 
panorama image, He is trying to offer an image generation method 
characterized by having processing which generates the new panorama 
image which will be generated from an image photoed and captured in a view 
location assumed out of the above-mentioned segment combining a cut-off 
partial image. 

[0015] Here, as an angle which sandwiches one image with which a path of 
the above-mentioned cylinder model is projected on the surface of the 
above-mentioned cylinder model is in agreement with a horizontal field 
angle of the above-mentioned camera, it is defined, and angle of rotation of 
the above-mentioned camera is defined as the body same in a ****** image is 
reflected. 

[0016] Thus, he processes a panorama image with which these images were 
continuously connected in two or more images which actually photoed and 
were captured, and is trying to generate a middle panorama image from a 
processed panorama image further in an image generation method of this 
invention. And in case a view change is made and a new panorama image is 
generated, he is trying to combine a partial image cut out of these panorama 
images and a middle panorama image. 

[0017] Thereby, like the conventional technology, since it becomes 
unnecessary to combine the incorporated image itself, even if it uses a 
metering device at the time of photography and does not record a location of 
a camera on it, it can be managed. Moreover, what is necessary is just to 
come to take a photograph like the conventional technology, in at least two 
view locations, even if it does not take a photograph using migration 
equipment. Furthermore, since a limit about a successive range of a view or 
the direction of a look decreases by using a panorama image, flexibility of 
walk-through can be increased by the minimum photography. 
[0018] In addition, about a partial image, since it will be necessary to expand 
these partial images according to a view location of a cut-off panorama image 
or a middle panorama image in fact in case two or more partial images are 
combined, if it is made to be the image of a strip-of-paper-like field, 
combination will become easy. 

[0019] Moreover, processing which generates the above-mentioned middle 
panorama image in an image generation method of this invention The 1st 
step which divides this panorama image into a field equivalent to each of two 
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or more bodies reflected in it about each of the above-mentioned panorama 
image, The 2nd step which defines attribute information which shows 
sequence of nearness of distance from a view location which corresponds for 
every field divided at the 1st step of the above, Inside of a middle panorama 
image which will be generated from an image photoed and captured in a view 
location assumed on a segment which connects between a view location 
which was divided at the 1st step of the above, and which corresponds from 
an image of this field for every field, and view locations corresponding to 
other panorama images, Superposition composition is carried out in 
sequence that attribute information which the 2nd step of the above defined 
shows an image of a field generated at the 3rd step which generates an 
image of the same field as this field, and the 3rd step of the above, and it can 
have the 4th step which generates the above-mentioned middle panorama 
image. 

[0020] Here, processing which generates the above-mentioned middle 
panorama image can have further the 5th step which fills up a pixel value of 
a pixel which this suffered a loss using a pixel value of a pixel located in the 
perimeter, when there is a pixel which suffered a loss in a middle panorama 
image generated at the 4th step of the above. 

[0021] Thereby, an image of a field equivalent to each body can be 
independently transformed now. That is, even when using an image with 
which many bodies are intermingled, distortion which takes place when 
physical relationship of bodies changes does not occur, and a phenomenon in 
which image quality deteriorates by such extreme partial distortion can be 
prevented. 

[0022] In addition, what is necessary is just to give a mask value of a proper 
to this field about each of a divided field, in order to distinguish this field and 
a field outside of it. 

[0023] Moreover, processing which generates some new panorama images in 
an image generation method of this invention combining the 
above-mentioned partial image The 1st step which calculates an average 
lightness value of a pixel in this partial image about each of the 
above-mentioned partial image, Make it have the 2nd step which amends a 
pixel value of each pixel in this partial image about each of the 
above-mentioned partial image to a value which is in agreement with a mean 
value of an average lightness value which an average lightness value of a 
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pixel in this partial image calculated at the 1st step of the above, or The 1st 
step which calculates an average lightness value of a pixel in this partial 
image about each of the above-mentioned partial image, The 2nd step which 
amends a lightness value of a pixel located in the boundary section of these 
two partial images about each of a group of a partial image of two ****** to a 
mean value of an average lightness value calculated at the 1st step of the 
above, The 3rd step which asks for difference with a lightness value after 
amending at a lightness value and the 2nd step of the above of this pixel 
about each of a pixel in the two above-mentioned partial images, A value 
whose effect of difference for which it asked at the 3rd step of the above 
about each of a pixel in the two above-mentioned partial images according to 
distance between pixels located in the above-mentioned boundary section 
decreases is calculated, and it can enable it to have the 4th step which adds a 
calculated value to a pixel value of this pixel. 

[0024] Thereby, in case two or more partial images are combined, a joint of 
each partial image can be connected smoothly. That is, it can prevent an 
unnatural boundary occurring in a final output image. 
[0025] 

[Embodiment of the Invention] Hereafter, the gestalt of operation of this 
invention is explained with reference to a drawing. 

[0026] Drawing 3 is the block diagram showing the fundamental hardware 
configuration of the computer system which enforced the image generation 
method of this invention. 

[0027] the inside of drawing, and 300 - for a scanner and 330, as for a 
processor and 350, a camera and 340 are [ an image display device and 310 / 
a mouse and 320 / storage and 360 ] high speed buses. 
[0028] In drawing 3 , a camera 330 may be for photoing a photographic 
subject and incorporating a static image or a dynamic image, and there is, or 
there are. [ two or more ] [ one ] Moreover, a scanner 320 is for incorporating 
a static image. 

[0029] In addition, when the image captured with the camera 330 or the 
scanner 320 is digital data, this image is directly recorded on storage 350, 
but when it is analog data, after being changed into digital data through the 
A/D conversion equipment which is not illustrated, it is recorded on storage 
350. 

[0030] Moreover, in drawing 3 , an image display device 300 is for displaying 
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the image captured with the camera 330 or the scanner 320, and the image 
generated by the image generation method of this invention. 
[0031] In addition, since it is necessary to do an activity, looking at the target 
image in case an operator divides an image into two or more fields or a 
characteristic portion is matched between images so that it may mention 
later, the image generated in the activation process of the image processing 
for realizing the image generation method of this invention will also be 
displayed on an image display device 300. 

[0032] Moreover, in drawing 3 , a mouse 310 is for an operator to input 
various directions, for example, in case an operator matches a characteristic 
portion between images, it serves as signal transduction data medium 
between a computer system and an operator. 

[0033] Moreover, in drawing 3 , a processor 340 performs the image 

processing for realizing the image generation method of this invention using 

data and the program which are memorized by storage 350. 

[0034] Moreover, in drawing 3 , a high speed bus 360 combines these the 

blocks of each, and serves to transmit data etc. mutually. 

[0035] Drawing 4 is explanatory drawing showing the storing field secured in 

the storage region on storage 305. 

[0036] The storing field 405 where the inputted image is stored in the 
storage region 400 on storage 350, the storing fields 410*445 where the 
image into which this was processed is stored, and the storing field 450 
where the program is stored are secured. 

[0037] In addition, the contents stored in each storing field are described 
with explanation of the image processing concerning this invention. 
[0038] Drawing 1 is a flow chart which shows the processing outline of the 
image processing for realizing the image generation method of this invention. 
[0039] Now, in the image processing concerning this invention, first, it is step 
100 and an image is inputted. 

[0040] In detail, the inputted image (input image) is stored in the storing 
field 405 (please refer to drawing 4 .) secured in the storage region 400 on 
storage 350 at step 100. 

[0041] Here, photography should be made, as an input image shall be the 
digital data photoed and incorporated with the digital camera and the center 
of the lens of a digital camera and the axis of rotation of the tripod of a digital 
camera were in agreement. And at step 100, the sequential input of two or 
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more images which changed, photoed and incorporated the sense of a digital 
camera for every fixed angle of rotation in the same location is carried out. 
[0042] Moreover, at step 100, an image is inputted twice [ at least ]. That is, 
2 sets of images which photoed respectively and were captured in at least 
two different locations are inputted. 

[0043] Then, at step 105, about each of the group of the image inputted at 
step 100, two or more images belonging to this group are connected in order, 
and a wide angle-image (panorama image) with which the scene for 1 round 
centering on the camera station by the digital camera is reflected in the 
image of one sheet is generated. 

[0044] In detail, at step 105, first, as shown in drawing 5 , the cylinder model 
500 of a radius d (530) is assumed, the "photograph 1 (510)" which is the 1st 
input image is projected on this cylinder model 500, and "a photograph 1 
(510)" is stuck on the surface of the cylinder model 500. The coordinate 
transformation equation in this case is expressed by (several l). 
[0045] 

[Equation l] 

x r = d f sin(atan(x/d)) 

y' = y • x7x = y/x * (d * ein(atan(x/d) ) ) 

■ • ■ (Sci ) 

[0046] Here, x and y are the coordinates on the plane of "a photograph 1 
(510)", and they are x 1 and a coordinate when projecting y' on the cylinder 
model 500. However, the projective center should be made into the center of 
gravity O of the cylinder model 500 (center of the medial axis 540 of the 
cylinder model 500), and the center of the longitudinal direction of "a 
photograph 1 (510)" shall be in contact with the surface of the cylinder model 
500. 

[0047] And also about the "photograph 2 (520)" which is the 2nd input image, 
the same projection as "a photograph 1 (510)" is performed, and "a 
photograph 2 (520)" is stuck on the surface of the cylinder model 500. At this 
time, "a photograph 2 (520)" and "a photograph 1 (510)" are the images with 
which only fixed angle of rotation changed, photoed and incorporated the 
sense of a digital camera, and as the portion to which the same scene is 
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reflected laps, "a photograph 2 (520)" is stuck. 

[0048] Then, it is necessary to carry out the axis of rotation of a digital 
camera as [ be / in agreement with the medial axis 540 of the cylinder model 
500 ], and to make angle of rotation of a digital camera into an angle to which 
the scene same in a ****** image is reflected and, and the angle which 
sandwiches one image stuck on the surface of the cylinder model 500 needs 
to make the radius d of the cylinder model 500 (530) a value which is in 
agreement with the horizontal field angle of a digital camera. 
[0049] Similarly, one panorama image is generable from 1 set of input 
images by sticking the 3rd input image, the 4th input image, and - on the 
surface of the cylinder model 500 in order. 

[0050] In addition, step 105 can be skipped if the dedicated device (for 
example, slit camera) for taking a photographic panorama can be used. 
[0051] Moreover, although the panorama image is used as an image with 
which the scene for 1 round centering on the camera station by the digital 
camera is reflected in this invention, even if this range is a semicircle or a 
smaller range, it does not interfere. 

[0052] Thus, at step 105, the panorama image in at least two different 
locations is generated respectively, and these generated panorama images 
are stored in the storing field 410 (please refer to drawing 4 .) secured in the 
storage region 400 on storage 350. 

[0053] Then, this panorama image is divided into the image (field image) of 
two or more fields at step 110 according to the body reflected in the scene 
about each of two or more panorama images generated at step 105. 
[0054] In detail, step 110 is realizable by using the primary differential of an 
image, performing an alternative edge extract according to the directions 
which an operator inputs from a mouse 310, extending a boundary line, and 
surrounding a body in a closed region as stated to the "image segmentation 
method" given in JP,3-218581,A. 

[0055] Furthermore, at step 110, the mask value for distinguishing this field 
image is given to each pixel in this field image about each of two or more 
divided field images, and two or more field images (mask image) after giving 
a mask value to the storing field 415 (refer to drawing 4 .) secured in the 
storage region 400 on storage 350 are stored. 

[0056] The panorama image 600 and size in every direction can be the same 
image, and this mask image can be data in which a mask value is shown 
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instead of the data (RGB data) of each pixel [ in / for the data which is each 
pixel / the panorama image 600 ], as shown in 610 of drawing 6 . [ as / each of 
that pixel of whose is the same data length (for example, 8 bits) as the data 
length of each pixel of the panorama image 600 ] 

[0057] And as shown in drawing 6 , when the panorama image 600 is a 
panorama image generated from the group of an input image which took a 
photograph of the interior of a room on the spot, [ for example, ] "1" is given 
as a mask value corresponding to the field image 650 with which the door 
630 is reflected. "2" is given as a mask value corresponding to the field image 
655 with which the aperture 635 is reflected. "3" is given as a mask value 
corresponding to the field image 660 with which the front wall 640 is 
reflected. "4" is given as a mask value corresponding to the field image 665 
with which the foliage plant 605 is reflected. "5" is given as a mask value 
corresponding to the field image 670 with which the wall 615 of a right hand 
is reflected, "6" is given as a mask value corresponding to the field image 645 
with which the wall 690 of a left hand is reflected, and "7" is given as a mask 
value corresponding to the other field images 685. 
[0058] Then, step 115 defines the attribute information showing the 
sequence of superposition composition on step 135 mentioned later to each of 
two or more field images divided at step 110. 

[0059] Since the context has arisen in the location of the body reflected in it 
when generated from the image which the panorama image took a 
photograph of on the spot, step 115 defines the sequence which shows the 
nearness of the distance from the camera station of each body as attribute 
information. 

[0060] And as further shown in drawing 6 , for every mask value given at 
step 110, the attribute table 620 for matching the attribute information 680 
defined to the field image which gave this mask value is created, and the 
attribute table 620 created in the storing field 420 (refer to drawing 4 .) 
secured in the storage region 400 on storage 350 is stored. 
[0061] In addition, even if it is a different mask value, the attribute 
information on the same value is to be defined as the field image 
corresponding to the body with the same distance from a camera station in 
the attribute table 620 in the example shown in drawing 6 , since a different 
mask value for every field image is given, but if the same mask value is given 
to the field image corresponding to the body with the same distance from a 
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camera station, attribute information will not overlap. 

[0062] Moreover, although the operator is made to perform division of a field, 
and the definition of attribute information, when it is a camera with the 
camera able to measure the depth information on the pixel value written in 
each element of the interior used for photography here, it is also possible for 
it to be made to perform division of a field and the definition of attribute 
information automatically from the measured depth information. 
[0063] Then, at step 120 and step 125, each pixel in this field image is moved 
where to each of two or more field images divided at step 110, while defining 
how the whole panorama image is made distorted, you defined, and make it 
distorted and distortion is generated according to a way. 

[0064] The method of specifying this portion matched between images by the 
segment can be used, and the amount of distortion of each pixel required 
since a distortion field is generated in an image is calculated based on each 
specified segment as you make it distorted and it is stated to "Feature-Based 
Image Metamorphosis" as the definition method of a way, for example. In 
addition, about the pixel which is not on the specified segment, that amount 
of distortion is computed according to the length of the foot of perpendicular 
given to the distance and the segment to each segment in the case of this 
count. 

[0065] In detail now, at step 120 and step 125 First, an operator sets in the 
"panorama image 1 (700)" and the "panorama image 2 (710)" which are 
shown in drawing 7 . The storing field 430 (please refer to drawing 4 .) 
secured in the storage region on storage 350 when the segment 720, a 
segment 740 and a segment 725, a segment 745 and a segment 730, and the 
segment 750 were specified as a portion matched respectively The 
correspondence relation of the specified segment is stored. 
[0066] In addition, it is desirable to match a panorama image and its camera 
station and to store in this storing field 430 further. 

[0067] Then, movement vector D which is the amount of distortion of each 
pixel in this panorama image in the case of deforming into another 
panorama image from this panorama image is calculated by (several 2) about 
each of the "panorama image 1 (700)" and the "panorama image 2 (710)", and 
calculated movement vector D is stored in the storing field 425 (please refer 
to drawing 4 .) secured in the storage region 400 on storage 350. 
[0068] The set of movement vector D which this movement vector D was 



15/31 



Japanese Publication number : 10-208074 



expressed by the coordinate value on image space, made it actual in this way, 
and was calculated about each pixel will express the field of the distortion at 
the time of transforming an image. 
[0069] 

[Equation 2] 



[0070] however, the vector to which P and Q make an end point the both ends 
of a segment 720 specified in "the panorama image 1 (700)", P', and Q - 1 - 
Vector and X f to which the vector which makes an end point the both ends of 
a segment 740 specified in "the panorama image (710)", and X make an end 
point the location of the pixel in the "panorama image 1 (700)" which is the 
object which calculates movement vector D shows the vector which makes 
the location after moving X an end point. 

[0071] Moreover, Perpendicular ( ) is the same length as the vector in ( ), and 
expresses a vector perpendicular to the vector in ( ). Moreover, the parameter 
expressed with the small letter expresses scalar quantity. 
[0072] In addition, when there are two or more specified segments, the 
weight average of the location of the corresponding points decided from each 
segment is carried out. That is, it is made to set the value which **(ed) the 
sum of the value WiDi which multiplied by the weight Wi which becomes so 
small that it becomes so large that the length of a segment is long and the 
distance to a segment becomes large at each movement vector Di which 
asked for and asked for the movement vector Di from each segment i by the 
sum of Wi to movement vector D. Movement vector D which can be found 
with this weighted mean can be expressed with (several 3). 
[0073] 

[Equation 3] 



[0074] In addition, by previous reference, Wi which becomes settled by 
(several 4) is used. 
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[0075] 

[Equation 4] 



[0076] However, length is [ the distance to Segment i, and a, b and p of the 
length of Segment i and dist ] constants. 

[0077] Thus, the processing which is made to generate the field of distortion 
and transforms an image is called "morphing." 

[0078] In addition, there is a method of an image making it distorted and 
defining a way etc. by making it distorted and moving the lattice point stuck 
on the image to the definition method of a way. That is, all the lattice points 
on an image are defined as, for example, corresponding to the specific lattice 
point on image with a certain another lattice point. If it does so, all the points 
in an on-the-spot photo image (pixel) are movable with the transformation 
which carries out geometric conversion of the quadrangle surrounded in the 
four lattice points on an image at the quadrangle on a corresponding image. 
Furthermore, a high order type may express transformation so that it may 
be connected smoothly. Thus, the generation method of distortion is not 
limited to "morphing." 

[0079] Then, the relative view location of the observer (an operator and the 
same person may also be another person objects.) when being based on the 
camera station by the digital camera at step 130 and the direction of a look 
which shows which direction the observer is looking at are inputted. 
[0080] In detail, at step 130, while an operator can, for example, specify the 
location on the map displayed on the image display device 300 using a mouse 
310 and displaying the image in a default location on an image display device 
300, it moves forward to this default location, or assignment that it is 
suitable rightward can be carried out. 

[0081] Then, a new panorama image is generated by carrying out 
superposition composition at step 135 according to the attribute information 
which defined each field image which transformed each of the field image 
divided at step 110, and deformed by step 115 based on the amount of 
distortion of each pixel computed at step 125 (movement vector D). 
[0082] In addition, about the details of step 135, it mentions later using the 
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flow chart shown in drawing 2 . 

[0083] Finally, the new panorama image generated at step 135 is expressed 
to an image display device 300 as step 140. 

[0084] Drawing 2 is a flow chart which shows detailed processing of step 135. 
[0085] As shown in drawing 2 , at step 200 (it is equivalent to step 130 of 
drawing 1 .), the relative view location of the observer when being based on 
the camera station by the digital camera and the direction of a look which 
shows which direction the observer is looking at are inputted. 
[0086] Of course, instead of the relative view location of the observer when 
being based on a camera station A Y-axis is set as the straight line which 
intersects perpendicularly with the X-axis and its straight line the straight 
line which connects two camera stations, for example although the absolute 
coordinate of a camera station and the absolute coordinate of an observer's 
location may be made to be inputted. As an X shaft-orientations component 
The internal ratio on the segment which connects two camera stations can be 
taken, and the movement magnitude of Y shaft orientations can be inputted 
as a Y-axis component by the relative distance which set distance between 
camera stations to "1." 

[0087] Moreover, although bearing of north, south, east and west may be 
inputted about the direction of a look, the method of the input of what times 
on the basis of the direction which has the specified substance, such as 
television, for example as it is an indoor case is also possible from there. 
[0088] Now, at least two panorama images are now generated by step 105 of 
drawing 1 . Furthermore, the amount of distortion of each pixel for being 
involved one side, while deforming about these two panorama images, by 
step 125 of drawing 1 , (movement vector D) is computed. 
[0089] So, at step 135, it is determined by making which internally dividing 
point into the mid-position whether the panorama image (middle panorama 
image) in this mid-position should be generated among the internally 
dividing points on the segment which connects with step 205 first two 
camera stations where two panorama images were obtained respectively. 
[0090] The decision of this mid-position is realizable by assigning at equal 
intervals according to the information which shows first the angle of 
visibility defined beforehand on the segment which connects two camera 
stations where two panorama images were respectively obtained in the 
mid-position of the number which determined the number of the 
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mid-position, then was determined. 

[0091] For example, as shown in drawing 8 , when using the panorama 
image in a camera station 800 as "the panorama image 1" and using the 
panorama image in a camera station 810 as "the panorama image 2", The 
middle panorama image in the mid-position 820 supposing the mid-position 
820 on the segment 830 which connects two camera stations 800,810 is 
determined (it is the panorama image which will be obtained when a digital 
camera is in the mid-position 820.) It is generable from two panorama 
images, the "panorama image 1" and the "panorama image 2." 
[0092] Moreover, for example, the panorama image 700 is the "panorama 
image 1" in a camera station 800, the panorama image 710 is the "panorama 
image 2" in a camera station 810 in drawing 9 , and the panorama image 900 
is a middle panorama image in the mid-position 820. Here, the example 
whose determined mid-position 820 is the middle point of a segment to which 
between the location of the door 910 in the panorama image 700 and the 
locations of the door 920 in the panorama image 710 is connected is shown, 
and as the location of the door 930 in the middle panorama image 900 
becomes in the middle of the location of the door 910 in the panorama image 
700, and the location of the door 920 in the panorama image 710, the middle 
panorama image 900 is generated. 

[0093] Then, it determines from the information which shows the angle of 
visibility which appointed beforehand the range displayed on an image 
display device 300 among the panorama images (that is, it is the new 
panorama image which should be generated.) which will be obtained when a 
digital camera is in the view location inputted at step 200 at step 210. 
[0094] It is because it is more efficient to generate only the portion 
equivalent to the display rectangle of a new panorama image since the range 
which can be displayed at a time is restricted to the image display device 300 
among the generated new panorama images even if this generated the 
middle panorama image in the mid-position determined at step 205 and it 
generated a new panorama image based on these middle panorama images. 
[0095] Then, at step 215, first, the middle panorama image in the 
mid-position determined at step 205 is generated, and about each of the 
generated middle panorama image, in order to generate the new panorama 
image of the display rectangle determined at step 210, the image of which 
sense of these middle panorama images asks for whether it is necessity. 
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[0096] In addition, the amount of distortion for generating a middle 
panorama image can be calculated from movement vector D according to the 
mid-position. 

[0097] Moreover, in case a middle panorama image is generated, the field 
image divided at step 110 of drawing 1 is transformed based on the amount 
of distortion, and it is made to carry out superposition composition in the 
sequence according to the attribute information which defined the field 
image after deformation by step 115 of drawing 1 . That is, since the 
attribute information on such a small value that it is close to a digital 
camera is defined, the value of attribute information puts the field image 
after deformation here on descending, i.e., the order which exists far away. 
[0098] For example, as shown in drawing 10 , in order to newly generate the 
panorama image 1005 which is the panorama image 1005 in a location 1000, 
and turned to the direction of the front, suppose that the middle panorama 
image in the mid-position 1030*1060 is generated besides the panorama 
image 700 in a camera station 1010, and the panorama image 710 in a 
camera station 1020. 

[0099] It turns out that expansion processing can be carried out and the 
image of the portion which is equivalent to the sense 1070 among the 
panorama images 1005 in a location 1000 by using the optical property "the 
same direction 1070 in a location 1010 can also observe the scene which is 
visible to the sense 1070 in a location 1000" here can generate the image of 
the portion equivalent to the sense 1070 of the panorama images 700. That is, 
about the panorama image 700, it turns out that it is required in order that 
the image of the portion equivalent to the sense 1070 may generate the 
panorama image 1005 in a location 1000. 

[0100] Similarly, respectively, if the image of the portion equivalent to the 
sense to the panorama image 1005 in a location 1000 is expanded for a 
suitable scale factor, the panorama image 1005 in a location 1000 is 
generable also about the other panorama image 1020 and the other middle 
panorama images 1030-1060, in false. 

[0101] Theoretically, since it is only one, only a number with the considerable 
panorama image which becomes a radical is needed, but even if the location 
on the panorama image whose sense corresponds makes the image of the 
field of the shape of a strip of paper with a certain amount of width of face 
correspond to this sense, it is satisfactory practically. Of course, the middle 
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panorama image in the mid-position is not what was restricted to four, if a 
large number, the generated panorama image will become a thing near an 
actual scene a certain forge fire, and the image quality when connecting the 
image of a strip -of-paper-like field also becomes good. 

[0102] The image for which it asks at step 215 then, in fact At step 220 which 
is the image (strip-of-paper-like field image) of a strip-of-paper-like field, 
makes, and continues The storing field 435 (please refer to drawing 4 .) 
which cut off the strip -of-paper-like field image for which it asked at step 215, 
and was secured in the storage region 400 on storage 350 about each of the 
middle panorama image generated at step 215 The cut-off strip-of-paper-like 
field image is stored. 

[0103] In addition, expansion processing of a scale factor with this behind 
suitable strip-of-paper-like field image is performed. 

[0104] For example, as shown in drawing 11 , when setting distance between 
two camera stations to 2L and preparing the mid-position of n pieces at equal 
intervals between them, the distance between the ****** location 1040 and 
1050 is expressed with l=2L/(n+l). 

[0105] Thus, when assuming that the middle panorama image is arranged at 
every distance 1, The panorama image 1005 in a location 1000 is divided like 
the strip-of-paper-like fields 1120-1150. The image of each strip-of-paper-like 
field about the strip-of-paper-like field 1120 It generates from the 
strip-of-paper-like field image cut out of the panorama image 700 in a 
location 1010. About the strip-of-paper-like field 1130 It generates from the 
strip-of-paper-like field image cut out of the middle panorama image in the 
mid-position 1030. About the strip-of-paper-like field 1140 It can generate 
from the strip-of-paper-like field image cut out of the middle panorama 
image in the mid-position 1040, and can generate about the 
strip-of-paper-like field 1150 from the strip-of-paper-like field image cut out 
of the middle panorama image in the mid-position 1050. 
[0106] As mentioned above, by step 220 from each of a panorama image and 
a middle panorama image Since the strip-of-paper-like field image required 
in order to generate the portion equivalent to the display rectangle of the 
panorama image determined at step 210 is cut off, at continuing step 230 By 
arranging these strip-of-paper-like field images in order, it is a new 
panorama image (it is a portion equivalent to a display rectangle in fact.). 
The storing field 440 (please refer to drawing 4 .) which generated and was 
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secured in the storage region 400 on storage 350 The generated new 
panorama image is stored. In addition, it is made to perform expansion 
processing of the suitable scale factor for each strip -of-paper-like field image 
to coincidence at this time. 

[0107] As shown in drawing 12 , in order to generate the new panorama 
image 1005 in a location 1220 now, the strip-of-paper-like field image which 
the strip-of-paper-like field image cut out of the panorama image 700 in a 
location 1230 is the strip-of-paper-like field image 1120, and cut out from the 
panorama image 1050 in a location 1240 presupposes that it is the 
strip-of-paper-like field image 1150. 

[0108] The sense to the new panorama image 1005 the panorama image 1050 
Although it is the panorama image which is in agreement with the direction 
of a look in a location 1220, therefore both three locations 1220, 1230, and 
1240 will be located at three top-most vertices which make a location 1240 
the right angle top-most vertices of a right triangle Here, the distance 
between a location 1230 and a location 1240 is LI, and suppose that the 
distance between a location 1240 and a location 1220 is L2. 
[0109] Furthermore, when the strip-of-paper-like field image 1150 expands 
some panorama images 1050 by S times, the strip-of-paper-like field image 
1120 should just expand some panorama images 700 by scale-factor S ! of 
(several 5). 
[0110] 

[Equation 5] 



[0111] Here, the value becomes large, so that the value of S is defined 
according to the movement magnitude to the view location inputted at step 
130 from the current view location and movement magnitude becomes large. 
[0112] Finally, inverse transformation of (several l) is performed to a new 
panorama image at step 235, a new panorama image is returned to the usual 
image by projecting on a view plane, and the image (projection image) 
returned to the storing field 445 (refer to drawing 4 .) secured in the storage 
region 400 on storage 350 is stored. 

[0113] Furthermore, the scene when changing a view into a user can be 
shown by transmitting this projection image to an image display device 300. 
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[0114] As explained above, according to the operation gestalt of this 
invention, the image which will be obtained when a photograph is taken in 
the assumed camera station using two or more images which photoed 
respectively and were captured in a different camera station is newly 
generable. 

[0115] Furthermore, in this case, one body was not only observed from 
various views, but some bodies are reflected to the image, and even when 
those physical relationship interchanges with an image, an image without 
sense of incongruity can be generated. 

[0116] Moreover, if there are at least two panorama images in a camera 
station which is different according to the operation gestalt of this invention, 
view modification in walk-through etc. is realized and the inside of the plane 
near a camera station can be observed to compensate for interactive 
actuation. 

[0117] therefore , without need the special equipments ( a metering device 
and the migration equipment of a camera parameter ) which needed the 
image a photograph of be took on the spot with the conventional technology 
in virtual reality systems , such as a presentation system used as the base , 
view modification in walk-through etc. be realize and it become possible to 
observe the image in various view locations . 

[0118] If the middle panorama image generated at step 215 is stored in 
storage 350, in case another panorama image will newly be generated after 
next time, this is used and it becomes unnecessary in addition, to regenerate 
in the operation gestalt explained above about the middle panorama image 
which already generates and is stored. 

[0119] Moreover, even if it does not generate the whole middle panorama 
image, you may make it generate only the portion (middle panorama image 
of a range required in order to generate the portion equivalent to the display 
rectangle of a new panorama image) equivalent to a strip-of-paper-like field 
image in the operation gestalt explained above, although the whole middle 
panorama image used as the radical which cuts off a strip-of-paper-like field 
image is generated. 

[0120] The ** which does not actually generate the middle panorama image 
in the mid-position determined at step 205 of drawing 2 at step 215 of 
drawing 2 in doing in this way, In order to generate the new panorama 
image of the display rectangle determined at step 210 of drawing 2 about 
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each of this middle panorama image While the image (strip-of-paper-like 
field image) of which sense of these middle panorama images asks for 
whether it is necessity, the field image and the amount of distortion which 
are needed in order to generate the strip-of-paper-like field image for which 
it asked are chosen. 

[0121] And the strip-of-paper-like field image which are some middle 
panorama images in the mid-position determined at step 205 of drawing 2 is 
generated at step 220 of drawing 2 using the field image and the amount of 
distortion which were chosen at step 215 of drawing 2 . 
[0122] In addition, although it can realize by transforming the field image 
chosen at step 215 of drawing 2 based on the amount of distortion chosen at 
step 215 of drawing 2 in case a strip-of-paper-like field image is generated, it 
is made to carry out superposition composition of the field image after 
deformation in the sequence according to the attribute information which 
step 115 of drawing 1 defined like **** at this time. 

[0123] Furthermore, the operation gestalt explained above can be used for 
various presentation systems, such as a walk-through system in a residence, 
and a travel experience system. 

[0124] For example, in using for the walk-through system in a residence, it 
generates two or more panorama images from two or more images which 
photo the inside of a residence in many camera stations, and were 
beforehand captured in each camera station at this time. And while an 
operator divides each generated panorama image into two or more fields and 
gives attribute information, these panorama images are matched using a 
segment. 

[0125] Since the front preparation for using the walk-through system in a 
residence is completed by performing the process so far, an operator will 
input the view location and the direction of a look of a customer in front of 
the customer who is an observer, if front preparation is completed. 
[0126] By this, a middle panorama image will be generated from the 
panorama image currently generated with front preparation, a 
strip-of-paper-like field image will be cut out from the generated middle 
panorama image and the panorama image already generated, and the new 
panorama image according to the view location and the direction of a look in 
which these strip-of-paper-like field images were put together and inputted 
will be generated further. Since the portion in which the generated new 
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panorama image is equivalent to the display rectangle at an image display 
device 300 is displayed, a customer only looks at the contents of a display of 
an image display device 300, and becomes possible [ observing the same 
scene as the scene observed while he is actually walking along the inside of a 
residence ]. 

[0127] By the way, in the operation gestalt explained above, it is still more 
desirable to perform processing which adjusts the color tone of a 
strip-of-paper-like field image so that it may state below. 
[0128] Drawing 13 is explanatory drawing showing the processing which 
generates a new panorama image combining the strip-of-paper-like field 
image generated respectively from two or more panorama images and middle 
panorama images. 

[0129] In drawing 13 , 1300, 1305, and 1310 are the strip-of-paper-like field 
images generated from the separate panorama image or the middle 
panorama image respectively. 

[0130] Generally, if the average lightness of two or more fields is 
independent, therefore each strip-of-paper-like field image is put in order as 
it is as shown in drawing 13 , the boundary section will look clearly. 
[0131] Then, since the lightness of the boundary section is connected 
smoothly, it is made to perform processing which adjusts the color tone of a 
strip-of-paper-like field image. 

[0132] As shown in the graph of drawing 13 , namely, the lightness average 
of each pixel in the strip-of-paper-like field image 1300 It will come, 
supposing it is the value shown in 1325, the lightness average of each pixel 
in the strip-of-paper-like field image 1305 is a value shown in 1330 and the 
lightness average of each image in the strip-of-paper-like field image 1310 is 
a value shown in 1335. While two strip-of-paper field images amend the pixel 
value of the pixel located in the ****** boundary section to the mean value of 
these lightness averages The pixel value of the pixel located near the 
boundary section is amended so that it may become the lightness value 
which the strip-of-paper-like field image originally had smoothly, as it keeps 
away from the boundary section. 

[0133] By performing such processing, it can prevent an unnatural boundary 
occurring in the panorama image and the final output image which were 
generated. 

[0134] Moreover, in the operation gestalt explained above, it is desirable to 
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perform processing which fills up the deficit section which happens when 
generating a middle panorama image further so that it may state below. 
[0135] Drawing 14 is the deficit section which happens when generating a 
middle panorama image from two panorama images, and explanatory 
drawing showing the processing which fills up this deficit section. 
[0136] Now, the case where the middle panorama image in the mid-position 
1420 is generated is considered from two panorama images generated from 
two or more images which photoed respectively and were captured in two 
locations 1415 and 1425. A place hides and is not visible to a body 1410 from 
a location 1415, and the body 1430 which exists in the scene used as the 
candidate for photography does not hide and have it from a location 1425. 
[ visible to a body 1405 ] Since the image of the body 1430 which has not been 
reflected to ** and the image photoed and captured in which locations 1415 
and 1425, either can interpolate two panorama images and cannot make 
them if it is **, the portion serves as the deficit section. 
[0137] Then, the processing which fills up the deficit section produced in 
such a reason is explained using the flow chart of drawing 14 . 
[0138] First, the deficit section is detected at step 1440. If decision whether 
the deficit arose or not stands the bit of the field when the field which 
assigned 1 bit is prepared for each pixel with an object image and the same 
size and a value is written in each pixel, it can make judgment that the pixel 
the bit does not stand is the deficit section. 

[0139] Then, at step 1445, the pixel located in the perimeter of the deficit 
section is investigated, by step 1450, the concentration value of the pixel 
which should be filled up is calculated and the deficit section is restored from 
the pixel value of the pixel investigated at step 1445. 

[0140] Then, the pixel set as the object of recoverability is moved at step 1455. 

For example, the location of an object pixel is moved so that it may become 

clockwise about deficit section inner circumference. Processing is ended, 

when return and all the deficit sections are filled up by step 1445 in order to 

repeat these processings until the deficit section is lost. 

[0141] By performing such processing, the deficit section which happens 

when generating a middle panorama image can be filled up. 

[0142] Moreover, in the operation gestalt explained above, it becomes 

possible by using a table to perform processing which projects a panorama 

image on a plane at step 235 of drawing 2 at a high speed so that it may state 
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below. 

[0143] Drawing 15 is explanatory drawing showing the table used in order to 
perform processing which projects a panorama image on a plane at a high 
speed. 

[0144] As shown in drawing 15 , this table 1500 consists of a pixel location (x 
y) 1520 whether the data corresponding to this pixel should be read from the 
pixel of the panorama image 1510 throat during projection about each of the 
pixel of the image after projection (it is the image displayed on an image 
display device 300.). 

[0145] The high-speed projection to a plane is attained by using this table 
1500, without carrying out geometric count. 

[0146] Moreover, although the cylinder model 500 is used in order to 
generate a panorama image, you may make it use a ball model in the 
operation gestalt explained above, so that it may state below. 
[0147] Drawing 16 is explanatory drawing showing the geometric conversion 
at the time of projecting a photograph on a ball model, turns one image 
(photograph) 1600 to the center 1620 of a ball, and projects it on the ball 
surface 1610. 

[0148] When the cylinder model 500 shown in drawing 5 was used, the axis 
of rotation of a digital camera had to be made in agreement with the medial 
axis 540 of the cylinder model 500, but if it fixes even the center of a digital 
camera in using a ball model, the image photoed with the free sense can be 
stuck. 

[0149] Thus, if the image projected on the ball model is used instead of the 
panorama image mentioned above, it can change the sense of a look in the 
vertical direction, and the sense of a look is not only horizontally movable, 
but can see to it. 

[0150] Moreover, in the operation gestalt explained above, in case a new 
panorama image is generated so that it may state below, by photography, it 
is using properly the color value and the color value of a middle panorama 
image which were actually acquired, and the image quality after generation 
can be improved. 

[0151] Since the same pixel as the panorama image generated from the 
actually photoed image may exist in the new panorama image after 
generation even if it is the portion generated from the strip-of-paper-like 
field which cut out from the middle panorama image and was expanded 
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About such a pixel, if the original color value is reproduced using the color 
value of the pixel of a panorama image, the image quality of the new 
panorama image after generation can be improved. 
[0152] In case drawing 17 generates a new panorama image, it is 
explanatory drawing showing how to use properly the color value and the 
color value of a middle panorama image which were acquired by photography. 
[0153] The table 1710 is matched with each of the pixel of the panorama 
image 1700 as shown in drawing 17 . The value theta of the sense is recorded 
the red from whom this table 1710 serves as a color value, green, the 
component values R, G, and B of a blue color, the absolute location (x y) of 
the center of the digital camera at the time of photography, and on a 
two-dimensional plane. 

[0154] Then, in case a new panorama image is generated, when there is a 
pixel from the value of this x, and y and theta, that color value can be 
sampled. 

[0155] In addition, the table 1710 shows the data structure in the case of 
using the cylinder model 500, and comes to show the data structure in the 
case of using a ball model in a table 1720. 

[0156] Moreover, in the operation gestalt explained above, in case a new 
panorama image is generated, he is trying to stick the strip -of-paper -like 
field image generated from the panorama image and the middle panorama 
image, but a new panorama image can be automatically generated from two 
or more panorama images currently prepared beforehand so that it may 
state below. 

[0157] Drawing 18 is explanatory drawing showing the structure for 
generating a new panorama image automatically from two or more 
panorama images currently prepared beforehand. 

[0158] In drawing 18 , 1800 is a table for matching the coordinate on the 
imagination camera station where the new panorama image which should be 
generated will be obtained, and the imagination map showing an observer's 
successive range (x y), and the panorama image 1810 divided into two or 
more field images is respectively matched with the coordinate on this map (x 

y). 

[0159] This panorama image 1810 is beforehand divided into two or more 
field images like "field 1 (1820)" - "a field 4 (1840)", and attribute data 
1850-1880 is further matched with each field image. 
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[0160] For example, about which panorama image of two or more panorama 
images prepared beforehand, which range in this panorama image is 
expanded to "a field 1 (1820)" with what kind of dilation ratio, and the 
attribute data 1850*1880 in which it is shown what kind of color correction 
should be performed is matched with it. 

[0161] If the imagination camera station where the new panorama image 
which should be generated will be obtained by this is decided, a desired 
panorama image can newly be automatically generated from two or more 
panorama images prepared beforehand. 

[0162] In addition, two or more panorama images prepared beforehand may 
be panorama images generated from the image which all actually photoed 
and captured, and may be the panorama image generated from the actually 
photoed image which was captured, and a middle panorama image generated 
from these. 

[0163] Furthermore, if in the case of the former a photograph is taken to 
coincidence in two or more camera stations and a panorama image is 
captured using the slit camera etc., it becomes possible to generate a new 
panorama image in parallel to photography, and even if the panorama image 
to capture is a dynamic image, it will become possible immediately to make a 
new panorama dynamic image. 
[0164] 

[Effect of the Invention] As explained above, according to this invention, the 
image which will be obtained when a photograph is taken in the assumed 
camera station using the image which photoed respectively and was 
captured in at least two different camera stations is newly generable. 
[0165] Furthermore, in this case, one body was not only observed from 
various views, but some bodies are reflected to the image, and even when 
those physical relationship interchanges with an image, an image without 
sense of incongruity can be generated. 

[0166] Moreover, if there are at least two panorama images in a camera 
station which is different according to this invention, view modification in 
walk-through etc. is realized and the inside of the plane near a camera 
station can be observed to compensate for interactive actuation. 
[0167] therefore , without not need the special equipments ( a metering 
device , migration equipment , etc. ) which needed the image a photograph of 
be took on the spot with the conventional technology in virtual reality 
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systems , such as a presentation system used as the base , or prepare many 
images needed with the conventional technology , view modification in 
walk-through etc. be realize and it become possible to observe the image in 
various view locations . 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing l] The flow chart showing the processing outline of the image 
processing for realizing the image generation method of this invention. 
[Drawing 2] The flow chart showing detailed processing of step 135 of 
drawing 1 . 

[Drawing 3l The block diagram showing the fundamental hardware 
configuration of the computer system which enforced the image generation 
method of this invention. 

[Drawing 41 Explanatory drawing showing the storing field secured in the 
storage region on storage. 

[Drawing 51 Explanatory drawing at the time of generating a panorama 
image from 1 set of images. 

[Drawing 61 Explanatory drawing showing the structure of a mask image 
and an attribute table. 

[Drawing 71 Explanatory drawing at the time of matching two panorama 
images using a segment. 

[Drawing 81 Explanatory drawing at the time of making a middle panorama 
image. 

[Drawing 9l Explanatory drawing showing the property of a middle 
panorama image. 

[Drawing 101 Explanatory drawing at the time of generating the new 
panorama image which will be obtained in the location which is not on the 
segment which connects a camera station. 

[Drawing 11] Explanatory drawing showing the relation between a 
panorama image and a middle panorama image, and the new panorama 
image generated from these. 

[Drawing 12] Explanatory drawing showing the image deformation 
processing at the time of generating a new panorama image from a 
panorama image and a middle panorama image. 
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[Drawing 13] Explanatory drawing having shown the processing which 
generates a new panorama image combining the strip-of-paper-like field 
image generated respectively from two or more panorama images and middle 
panorama images. 

[Drawing 14] The deficit section which happens when generating a middle 

panorama image from two panorama images, and explanatory drawing 

showing the processing which fills up this deficit section. 

[Drawing 15] Explanatory drawing showing the table used in order to 

perform processing which projects a panorama image on a plane at a high 

speed. 

[Drawing 16] Explanatory drawing showing the geometric conversion at the 
time of projecting a photograph on a ball model. 

[Drawing 17] Explanatory drawing showing how to use properly the color 
value and the color value of a middle panorama image which were acquired 
by photography in case a new panorama image is generated. 
[Drawing 18] Explanatory drawing having shown the structure for 
generating a new panorama image automatically from the panorama image 
in two or more camera stations. 
[Description of Notations] 

300 [ Camera, ] - An image display device, 310 A mouse, 320 - A 
scanner, 330 340 [ - Storage region, ] A processor, 350 Storage, 360 - A 
high speed bus, 400 405 ■- An input image storing field, 410 - A panorama 
image storing field, 415 — Mask image storing field, 420 [ -■ A 
strip-of-paper-like field image storing field 440 / ■■ A new panorama image 
storing field, 445 / A projection image storing field, 450 / Program storage 
area. ] - An attribute table storing field, 425 - A movement vector storing 
field, 430 -- A correspondence relation storing field, 435 
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